Use of Detected Objects for Image Processing

ABSTRACT

Methods and systems for the use of detected objects for image processing are described. A computing device autonomously controlling a vehicle may receive images of the environment surrounding the vehicle from an image-capture device coupled to the vehicle. In order to process the images, the computing device may receive information indicating characteristics of objects in the images from one or more sources coupled to the vehicle. Examples of sources may include RADAR, LIDAR, a map, sensors, a global positioning system (GPS), or other cameras. The computing device may use the information indicating characteristics of the objects to process received images, including determining the approximate locations of objects within the images. Further, while processing the image, the computing device may use information from sources to determine portions of the image to focus upon that may allow the computing device to determine a control strategy based on portions of the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of U.S. patentapplication Ser. No. 16/714,065, filed on Dec. 13, 2019, which is acontinuation of U.S. patent application Ser. No. 15/713,263, filed onSep. 22, 2017 (now U.S. Pat. No. 10,509,402), which is a continuation ofU.S. patent application Ser. No. 14/855,009, filed on Sep. 15, 2015 (nowU.S. Pat. No. 9,804,597), which is a continuation of U.S. patentapplication Ser. No. 13/864,336, filed on Apr. 17, 2013 (now U.S. Pat.No. 9,164,511), all of which are hereby incorporated by reference intheir entirety including all tables, figures, and claims.

BACKGROUND

Autonomous vehicles use various computing systems to aid in transportingpassengers from one location to another. In some cases, autonomousvehicles may operate without receiving any input from a driver orpassenger, but through analyzing objects. Some autonomous vehicles mayuse a system of image-capture devices to capture images of theenvironment including objects around the autonomous vehicle. Anautonomous vehicle may use the images to assist in making determinationsduring operation. Some autonomous vehicles may require some initialinput or continuous input from an operator, such as a pilot, driver, orpassenger. Other systems, for example autopilot systems, may be usedonly when the system has been engaged, which permits the operator toswitch from a manual mode (where the operator exercises a high degree ofcontrol over the movement of the vehicle) to an autonomous mode (wherethe vehicle essentially drives itself) to modes that lie somewhere inbetween.

SUMMARY

The present application discloses examples that relate to the use ofdetected objects for image processing. In one aspect, the presentapplication describes a method. The method may comprise receiving, froman image-capture device coupled to a vehicle, an image comprising anobject. The method also may comprise receiving, via one or more sourcecoupled to the vehicle, information indicating one or morecharacteristics of the object. The method may further comprisedetermining by a computer device, using the information indicating oneor more characteristics of the object, a portion of the image depictingthe object. In addition, the method may include processing the portionof the image to determine a control strategy for the vehicle. The methodmay also comprise providing instructions to control the vehicle based onthe control strategy for the vehicle.

In another aspect, the present application describes a non-transitorycomputer readable medium having stored thereon executable instructionsthat, upon execution by a computing device, cause the computing deviceto perform functions. The functions may comprise receiving, from animage-capture device coupled to a vehicle, an image comprising anobject. In addition, the functions may comprise receiving, via one ormore sources coupled to the vehicle, information indicating one or morecharacteristics of the object. The functions may further comprisedetermining, using the information indicating one or morecharacteristics of the object, a portion of the image depicting theobject. Additionally, the functions may comprise processing the portionof the image to determine a control strategy for the vehicle andproviding instructions to control the vehicle based on the controlstrategy for the vehicle.

In still another aspect, the present application describes a controlsystem. The control system may comprise at least one processor. Thecontrol system also may comprise a memory having stored thereonexecutable instructions that, upon execution by the at least oneprocessor, cause the control system to perform functions comprisingreceiving, from an image-capture device coupled to a vehicle, an imagecomprising an object. The functions may include receiving, via one ormore sources coupled to the vehicle, information indicating one or morecharacteristics of the object. Further, the functions may includedetermining, using the information indicating one or morecharacteristics of the object, a portion of the image depicting theobject. In addition, the functions may include processing the portion ofthe image to determine a control strategy for the vehicle. The functionsmay also include providing instructions to control the vehicle based onthe control strategy for the vehicle.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a simplified block diagram of an example vehicle, inaccordance with an example embodiment.

FIG. 2 illustrates an example vehicle, in accordance with an exampleembodiment.

FIG. 3 is a flow chart of an example method for the use of detectedobjects in image processing.

FIG. 4A illustrates an example scenario for using detected objects inimage processing in order to determine if a vehicle is turning.

FIG. 4B illustrates an example scenario for using detected objects inimage processing in order to determine the potential actions of a nearbyvehicle.

FIG. 4C illustrates an example scenario for using detected objects inimage processing to determine if an object is partially occluded.

FIG. 4D illustrates an example scenario for using detected objects inimage processing to determine the state of a traffic signal.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise. The illustrative systemand method embodiments described herein are not meant to be limiting. Itmay be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

An autonomous vehicle operating on a road or other surface may rely onidentifying objects within the vicinity of the vehicle in order todetermine a safe trajectory or path for the vehicle to continuetraveling upon. A computing device controlling the vehicle in autonomousmode may use one or more image-capture devices, such as cameras or videorecorders, to capture images of objects within the surroundingenvironment of the vehicle. The image-capture devices may be positionedon a vehicle to capture the various angles of objects nearby the vehicleor within the upcoming path of the vehicle. The autonomous vehicle maybe configured to continuously capture images of the surroundingenvironment in real-time.

The computing device controlling the vehicle may be configured to useresources to locate objects, such as other vehicles, within the capturedimages. In some examples, a computing device may process the capturedimages utilizing additional information provided by other sourcescoupled to the vehicle, such as RADAR, LIDAR, maps, and additionalcameras. RADAR may use radio waves to determine a range, altitude,direction, or speed of objects. LIDAR is a light detection and rangingor laser imaging detection and ranging. The LIDAR system may use opticalremote sensing technology to measure the distance to, or otherproperties of targets by illuminating the target with laser light andanalyzing the backscattered light. The other sources may be associatedor attached to the vehicle and may be controlled by one or morecomputing devices. The additional information may includecharacteristics about specific objects within the environment of thevehicle. For example, a computing device may utilize the speed and sizeof objects as determined by a LIDAR system using lasers or sensors inorder to approximate the location of the particular objects withincaptured images. The computing device may also use RADAR or maps todetermine locations of objects relative to the vehicle, which may assistthe computing device in determining where the objects may approximatelybe located within an image captured of the surrounding environment. Thecomputing device may receive other information from sources that mayassist in image processing as well.

In some examples, a computing device operating a vehicle autonomouslymay use information from multiple sources to assist with detectingobjects within the environment of the vehicle and to process imagesreceived from an image-capture device. For example, a computing devicemay capture an image of a nearby vehicle traveling on the same road. Inorder to determine various information about the vehicle (e.g., the typeof vehicle or the model), the computing device may focus upon detailswithin the image to determine specific features of the nearby vehicle.To determine portions of the image containing details about the vehicle,the computing device may use information about the speed, size, andrelative location in the environment gathered by sources.

In some examples, the computing device may be configured to locate brakesignals or turn signals of other vehicles within images to determinewhether or not the vehicle is braking or preparing to turn. Likewise, acomputing device controlling a vehicle may use a similar process tolocate other objects within captured images, including but not limitedto pedestrians, traffic lights, signs, animals, and speed bumps, etc.

In another example, a computing device controlling a vehicleautonomously may receive information about nearby objects from sources,which may include the approximate height, width, or length of theobjects. For example, a system of lasers may be configured to measurethe sizes of nearby objects. Similarly, other sources may be capable ofdetermining the approximate distance between the autonomous vehicle andan object. The computing device may use the information to determine thelocation of particular objects within the captured images received fromthe image-capture device(s) coupled on the vehicle. Similarly, thecomputing device may receive information including the shapes, speed, ordistance relative to the vehicle controlled by the computing device fromthe other sources. Other additional information may be provided by thesources on objects within the vicinity of a vehicle.

In some implementations, a computing device may use one or moretransforms to determine data that allows the computing device todetermine or infer the location of the objects within an image capturedby an image-capture device. The transform may include a camera matrix,or other types of visual mapping from points in the real environment tothe image. For example, the computing device may use a transform todetermine the location of objects within an image by matching thelocation of points in the actual situation and the location of the samepoints within the image taken of the situation. Further, a computingdevice using transforms may factor in additional information receivedabout objects in the environment to locate objects within images.

In addition, the computing device may use information provided fromsources, such as LIDAR, RADAR, etc. to determine which objects arepresent in the environment but may be occluded within an image (notvisible). The computing device may determine possible occlusions usingpredetermined information about other known objects and/or informationobtained from the additional sources. For example, the computing devicemay analyze intersections between any estimated areas of the image thatanother object is positioned within and determine which object shouldappear in front of the other in such intersection cases. Additionally, acomputing device may also determine partial occlusions, such asdetermining that the left side of another vehicle is likely occluded inthe image since the right side is visible within captured images.

In some implementations, a computing device controlling a vehicle mayuse the location of objects in images as determined through usingcaptured images and additional sources (e.g., LIDAR, RADAR) to performfurther analysis on the objects or in order to determine a controlstrategy for operating the vehicle. For example, a computing device mayuse images received from an image-capture device of a nearby vehicle anduse information from additional sources to determine an approximatelocation of specific objects of the vehicle within the image. Thecomputing device may use the image processing to further determinefeatures of the vehicle, such as the license plate number, or if brakelights or a turn signal is being used. The computing device may furtherdetermine a control strategy based on information gathered from theimage processing. For example, the computing device may determine safepaths for travel depending on the information about surrounding objectsin the environment of the vehicle.

In some examples, the computing device may be configured to control thevehicle through a vehicle control system. An example vehicle controlsystem may be implemented in or may take the form of a vehicle.Alternatively, a vehicle control system may be implemented in or takethe form of other vehicles, such as cars, trucks, motorcycles, buses,boats, airplanes, helicopters, lawn mowers, recreational vehicles,amusement park vehicles, farm equipment, construction equipment, trams,golf carts, trains, and trolleys. Other vehicles are possible as well.

Further, an example system may take the form of a non-transitorycomputer-readable medium, which has program instructions stored thereonthat are executable by at least one processor to provide thefunctionality described herein. An example system may also take the formof a vehicle or a subsystem of a vehicle that includes such anon-transitory computer-readable medium having such program instructionsstored thereon.

I. Example Vehicle

Referring now to the Figures, FIG. 1 is a simplified block diagram of anexample vehicle 100, in accordance with an example embodiment.Components coupled to or included in the vehicle 100 may include apropulsion system 102, a sensor system 104, a control system 106,peripherals 108, a power supply 110, a computing device 111, and a userinterface 112. The computing device 111 may include a processor 113, anda memory 114. The computing device 111 may be a controller, or part ofthe controller, of the vehicle 100. The memory 114 may includeinstructions 115 executable by the processor 113, and may also store mapdata 116. Components of the vehicle 100 may be configured to work in aninterconnected fashion with each other and/or with other componentscoupled to respective systems. For example, the power supply 110 mayprovide power to all the components of the vehicle 100. The computingdevice 111 may be configured to receive information from and control thepropulsion system 102, the sensor system 104, the control system 106,and the peripherals 108. The computing device 111 may be configured togenerate a display of images on and receive inputs from the userinterface 112.

In other examples, the vehicle 100 may include more, fewer, or differentsystems, and each system may include more, fewer, or differentcomponents. Additionally, the systems and components shown may becombined or divided in any number of ways.

The propulsion system 102 may may be configured to provide poweredmotion for the vehicle 100. As shown, the propulsion system 102 includesan engine/motor 118, an energy source 120, a transmission 122, andwheels/tires 124.

The engine/motor 118 may be or include any combination of an internalcombustion engine, an electric motor, a steam engine, and a Sterlingengine. Other motors and engines are possible as well. In some examples,the propulsion system 102 could include multiple types of engines and/ormotors. For instance, a gas-electric hybrid car could include a gasolineengine and an electric motor. Other examples are possible.

The energy source 120 may be a source of energy that powers theengine/motor 118 in full or in part. That is, the engine/motor 118 maybe configured to convert the energy source 120 into mechanical energy.Examples of energy sources 120 include gasoline, diesel, otherpetroleum-based fuels, propane; other compressed gas-based fuels,ethanol, solar panels, batteries, and other sources of electrical power.The energy source(s) 120 could additionally or alternatively include anycombination of fuel tanks, batteries, capacitors, and/or flywheels. Insome examples, the energy source 120 may provide energy for othersystems of the vehicle 100 as well.

The transmission 122 may be configured to transmit mechanical power fromthe engine/motor 118 to the wheels/tires 124. To this end, thetransmission 122 may include a gearbox, clutch, differential, driveshafts, and/or other elements. In examples where the transmission 122includes drive shafts, the drive shafts could include one or more axlesthat are configured to be coupled to the wheels/tires 124.

The wheels/tires 124 of vehicle 100 could be configured in variousformats, including a unicycle, bicycle/motorcycle, tricycle, orcar/truck four-wheel format. Other wheel/tire formats are possible aswell, such as those including six or more wheels. The wheels/tires 124of vehicle 100 may be configured to rotate differentially with respectto other wheels/tires 124. In some examples, the wheels/tires 124 mayinclude at least one wheel that is fixedly attached to the transmission122 and at least one tire coupled to a rim of the wheel that could makecontact with the driving surface. The wheels/tires 124 may include anycombination of metal and rubber, or combination of other materials.

The propulsion system 102 may additionally or alternatively includecomponents other than those shown.

The sensor system 104 may include a number of sensors configured tosense information about an environment in which the vehicle 100 islocated. As shown, the sensors of the sensor system include a GlobalPositioning System (GPS) module 126, an inertial measurement unit (IMU)128, a radio detection and ranging (RADAR) unit 130, a laser rangefinderand/or light detection and ranging (LIDAR) unit 132, a camera 134, andactuators 136 configured to modify a position and/or orientation of thesensors. The sensor system 104 may include additional sensors as well,including, for example, sensors that monitor internal systems of thevehicle 100 (e.g., an 02 monitor, a fuel gauge, an engine oiltemperature, etc.). Other sensors are possible as well.

The GPS module 126 may be any sensor configured to estimate a geographiclocation of the vehicle 100. To this end, the GPS module 126 may includea transceiver configured to estimate a position of the vehicle 100 withrespect to the Earth, based on satellite-based positioning data. In anexample, the computing device 111 may be configured to use the GPSmodule 126 in combination with the map data 116 to estimate a locationof a lane boundary on road on which the vehicle 100 may be travellingon. The GPS module 126 may take other forms as well.

The IMU 128 may be any combination of sensors configured to senseposition and orientation changes of the vehicle 100 based on inertialacceleration. In some examples, the combination of sensors may include,for example, accelerometers and gyroscopes. Other combinations ofsensors are possible as well.

The RADAR unit 130 may be considered as an object detection system thatmay be configured to use radio waves to determine characteristics of theobject such as range, altitude, direction, or speed of the object. TheRADAR unit 130 may be configured to transmit pulses of radio waves ormicrowaves that may bounce off any object in a path of the waves. Theobject may return a part of energy of the waves to a receiver (e.g.,dish or antenna), which may be part of the RADAR unit 130 as well. TheRADAR unit 130 also may be configured to perform digital signalprocessing of received signals (bouncing off the object) and may beconfigured to identify the object.

Other systems similar to RADAR have been used in other parts of theelectromagnetic spectrum. One example is LIDAR (light detection andranging), which may be configured to use visible light from lasersrather than radio waves.

The LIDAR unit 132 may include a sensor configured to sense or detectobjects in an environment in which the vehicle 100 is located usinglight. Generally, LIDAR is an optical remote sensing technology that canmeasure distance to, or other properties of, a target by illuminatingthe target with light. As an example, the LIDAR unit 132 may include alaser source and/or laser scanner configured to emit laser pulses and adetector configured to receive reflections of the laser pulses. Forexample, the LIDAR unit 132 may include a laser range finder reflectedby a rotating mirror, and the laser is scanned around a scene beingdigitized, in one or two dimensions, gathering distance measurements atspecified angle intervals. In examples, the LIDAR unit 132 may includecomponents such as light (e.g., laser) source, scanner and optics,photo-detector and receiver electronics, and position and navigationsystem.

In an example, The LIDAR unit 132 may be configured to use ultraviolet(UV), visible, or infrared light to image objects and can be used with awide range of targets, including non-metallic objects. In one example, anarrow laser beam can be used to map physical features of an object withhigh resolution.

In examples, wavelengths in a range from about 10 micrometers (infrared)to about 250 nm (UV) could be used. Typically light is reflected viabackscattering. Different types of scattering are used for differentLIDAR applications, such as Rayleigh scattering, Mie scattering andRaman scattering, as well as fluorescence. Based on different kinds ofbackscattering, LIDAR can be accordingly called Rayleigh LIDAR, MieLIDAR, Raman LIDAR and Na/Fe/K Fluorescence LIDAR, as examples. Suitablecombinations of wavelengths can allow for remote mapping of objects bylooking for wavelength-dependent changes in intensity of reflectedsignals, for example.

Three-dimensional (3D) imaging can be achieved using both scanning andnon-scanning LIDAR systems. “3D gated viewing laser radar” is an exampleof a non-scanning laser ranging system that applies a pulsed laser and afast gated camera. Imaging LIDAR can also be performed using an array ofhigh speed detectors and a modulation sensitive detectors arraytypically built on single chips using CMOS (complementarymetal-oxide-semiconductor) and hybrid CMOS/CCD (charge-coupled device)fabrication techniques. In these devices, each pixel may be processedlocally by demodulation or gating at high speed such that the array canbe processed to represent an image from a camera. Using this technique,many thousands of pixels may be acquired simultaneously to create a 3Dpoint cloud representing an object or scene being detected by the LIDARunit 132.

A point cloud may include a set of vertices in a 3D coordinate system.These vertices may be defined by X, Y, and Z coordinates, for example,and may represent an external surface of an object. The LIDAR unit 132may be configured to create the point cloud by measuring a large numberof points on the surface of the object, and may output the point cloudas a data file. As the result of a 3D scanning process of the object bythe LIDAR unit 132, the point cloud can be used to identify andvisualize the object.

In one example, the point cloud can be directly rendered to visualizethe object. In another example, the point cloud may be converted topolygon or triangle mesh models through a process that may be referredto as surface reconstruction. Example techniques for converting a pointcloud to a 3D surface may include Delaunay triangulation, alpha shapes,and ball pivoting. These techniques include building a network oftriangles over existing vertices of the point cloud. Other exampletechniques may include converting the point cloud into a volumetricdistance field and reconstructing an implicit surface so defined througha marching cubes algorithm.

The camera 134 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which thevehicle 100 is located. To this end, the camera may be configured todetect visible light, or may be configured to detect light from otherportions of the spectrum, such as infrared or ultraviolet light. Othertypes of cameras are possible as well. The camera 134 may be atwo-dimensional detector, or may have a three-dimensional spatial range.In some examples, the camera 134 may be, for example, a range detectorconfigured to generate a two-dimensional image indicating a distancefrom the camera 134 to a number of points in the environment. To thisend, the camera 134 may use one or more range detecting techniques. Forexample, the camera 134 may be configured to use a structured lighttechnique in which the vehicle 100 illuminates an object in theenvironment with a predetermined light pattern, such as a grid orcheckerboard pattern and uses the camera 134 to detect a reflection ofthe predetermined light pattern off the object. Based on distortions inthe reflected light pattern, the vehicle 100 may be configured todetermine the distance to the points on the object. The predeterminedlight pattern may comprise infrared light, or light of anotherwavelength.

The actuators 136 may, for example, be configured to modify a positionand/or orientation of the sensors.

The sensor system 104 may additionally or alternatively includecomponents other than those shown.

The control system 106 may be configured to control operation of thevehicle 100 and its components. To this end, the control system 106 mayinclude a steering unit 138, a throttle 140, a brake unit 142, a sensorfusion algorithm 144, a computer vision system 146, a navigation orpathing system 148, and an obstacle avoidance system 150.

The steering unit 138 may be any combination of mechanisms configured toadjust the heading or direction of the vehicle 100.

The throttle 140 may be any combination of mechanisms configured tocontrol the operating speed and acceleration of the engine/motor 118and, in turn, the speed and acceleration of the vehicle 100.

The brake unit 142 may be any combination of mechanisms configured todecelerate the vehicle 100. For example, the brake unit 142 may usefriction to slow the wheels/tires 124. As another example, the brakeunit 142 may be configured to be regenerative and convert the kineticenergy of the wheels/tires 124 to electric current. The brake unit 142may take other forms as well.

The sensor fusion algorithm 144 may include an algorithm (or a computerprogram product storing an algorithm) executable by the computing device111, for example. The sensor fusion algorithm 144 may be configured toaccept data from the sensor system 104 as an input. The data mayinclude, for example, data representing information sensed at thesensors of the sensor system 104. The sensor fusion algorithm 144 mayinclude, for example, a Kalman filter, a Bayesian network, or anotheralgorithm. The sensor fusion algorithm 144 further may be configured toprovide various assessments based on the data from the sensor system104, including, for example, evaluations of individual objects and/orfeatures in the environment in which the vehicle 100 is located,evaluations of particular situations, and/or evaluations of possibleimpacts based on particular situations. Other assessments are possibleas well

The computer vision system 146 may be any system configured to processand analyze images captured by the camera 134 in order to identifyobjects and/or features in the environment in which the vehicle 100 islocated, including, for example, lane information, traffic signals andobstacles. To this end, the computer vision system 146 may use an objectrecognition algorithm, a Structure from Motion (SFM) algorithm, videotracking, or other computer vision techniques. In some examples, thecomputer vision system 146 may additionally be configured to map theenvironment, track objects, estimate speed of objects, etc.

The navigation and pathing system 148 may be any system configured todetermine a driving path for the vehicle 100. The navigation and pathingsystem 148 may additionally be configured to update the driving pathdynamically while the vehicle 100 is in operation. In some examples, thenavigation and pathing system 148 may be configured to incorporate datafrom the sensor fusion algorithm 144, the GPS module 126, and one ormore predetermined maps so as to determine the driving path for thevehicle 100.

The obstacle avoidance system 150 may be any system configured toidentify, evaluate, and avoid or otherwise negotiate obstacles in theenvironment in which the vehicle 100 is located.

The control system 106 may additionally or alternatively includecomponents other than those shown.

Peripherals 108 may be configured to allow the vehicle 100 to interactwith external sensors, other vehicles, and/or a user. To this end, theperipherals 108 may include, for example, a wireless communicationsystem 152, a touchscreen 154, a microphone 156, and/or a speaker 158.

The wireless communication system 152 may be any system configured to bewirelessly coupled to one or more other vehicles, sensors, or otherentities, either directly or via a communication network. To this end,the wireless communication system 152 may include an antenna and achipset for communicating with the other vehicles, sensors, or otherentities either directly or over an air interface. The chipset orwireless communication system 152 in general may be arranged tocommunicate according to one or more other types of wirelesscommunication (e.g., protocols) such as Bluetooth, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), Zigbee, dedicated short range communications (DSRC), and radiofrequency identification (RFID) communications, among otherpossibilities. The wireless communication system 152 may take otherforms as well.

The touchscreen 154 may be used by a user to input commands to thevehicle 100. To this end, the touchscreen 154 may be configured to senseat least one of a position and a movement of a user's finger viacapacitive sensing, resistance sensing, or a surface acoustic waveprocess, among other possibilities. The touchscreen 154 may be capableof sensing finger movement in a direction parallel or planar to thetouchscreen surface, in a direction normal to the touchscreen surface,or both, and may also be capable of sensing a level of pressure appliedto the touchscreen surface. The touchscreen 154 may be formed of one ormore translucent or transparent insulating layers and one or moretranslucent or transparent conducting layers. The touchscreen 154 maytake other forms as well.

The microphone 156 may be configured to receive audio (e.g., a voicecommand or other audio input) from a user of the vehicle 100. Similarly,the speakers 158 may be configured to output audio to the user of thevehicle 100.

The peripherals 108 may additionally or alternatively include componentsother than those shown.

The power supply 110 may be configured to provide power to some or allof the components of the vehicle 100. To this end, the power supply 110may include, for example, a rechargeable lithium-ion or lead-acidbattery. In some examples, one or more banks of batteries could beconfigured to provide electrical power. Other power supply materials andconfigurations are possible as well. In some examples, the power supply110 and energy source 120 may be implemented together, as in someall-electric cars.

The processor 113 included in the computing device 111 may comprise oneor more general-purpose processors and/or one or more special-purposeprocessors (e.g., image processor, digital signal processor, etc.). Tothe extent that the processor 113 includes more than one processor, suchprocessors could work separately or in combination. The computing device111 may be configured to control functions of the vehicle 100 based oninput received through the user interface 112, for example.

The memory 114, in turn, may comprise one or more volatile and/or one ormore non-volatile storage components, such as optical, magnetic, and/ororganic storage, and the memory 114 may be integrated in whole or inpart with the processor 113. The memory 114 may contain the instructions115 (e.g., program logic) executable by the processor 113 to executevarious vehicle functions, including any of the functions or methodsdescribed herein.

The components of the vehicle 100 could be configured to work in aninterconnected fashion with other components within and/or outside theirrespective systems. To this end, the components and systems of thevehicle 100 may be communicatively linked together by a system bus,network, and/or other connection mechanism (not shown).

Further, while each of the components and systems is shown to beintegrated in the vehicle 100, in some examples, one or more componentsor systems may be removably mounted on or otherwise connected(mechanically or electrically) to the vehicle 100 using wired orwireless connections.

The vehicle 100 may include one or more elements in addition to orinstead of those shown. For example, the vehicle 100 may include one ormore additional interfaces and/or power supplies. Other additionalcomponents are possible as well. In these examples, the memory 114 mayfurther include instructions executable by the processor 113 to controland/or communicate with the additional components.

FIG. 2 illustrates an example vehicle 200, in accordance with anembodiment. In particular, FIG. 2 shows a Right Side View, Front View,Back View, and Top View of the vehicle 200. Although vehicle 200 isillustrated in FIG. 2 as a car, other examples are possible. Forinstance, the vehicle 200 could represent a truck, a van, a semi-trailertruck, a motorcycle, a golf cart, an off-road vehicle, or a farmvehicle, among other examples. As shown, the vehicle 200 includes afirst sensor unit 202, a second sensor unit 204, a third sensor unit206, a wireless communication system 208, and a camera 210.

Each of the first, second, and third sensor units 202-206 may includeany combination of global positioning system sensors, inertialmeasurement units, RADAR units, LIDAR units, cameras, lane detectionsensors, and acoustic sensors. Other types of sensors are possible aswell.

While the first, second, and third sensor units 202 are shown to bemounted in particular locations on the vehicle 200, in some examples thesensor unit 202 may be mounted elsewhere on the vehicle 200, eitherinside or outside the vehicle 200. Further, while only three sensorunits are shown, in some examples more or fewer sensor units may beincluded in the vehicle 200.

In some examples, one or more of the first, second, and third sensorunits 202-206 may include one or more movable mounts on which thesensors may be movably mounted. The movable mount may include, forexample, a rotating platform. Sensors mounted on the rotating platformcould be rotated so that the sensors may obtain information from eachdirection around the vehicle 200. Alternatively or additionally, themovable mount may include a tilting platform. Sensors mounted on thetilting platform could be tilted within a particular range of anglesand/or azimuths so that the sensors may obtain information from avariety of angles. The movable mount may take other forms as well.

Further, in some examples, one or more of the first, second, and thirdsensor units 202-206 may include one or more actuators configured toadjust the position and/or orientation of sensors in the sensor unit bymoving the sensors and/or movable mounts. Example actuators includemotors, pneumatic actuators, hydraulic pistons, relays, solenoids, andpiezoelectric actuators. Other actuators are possible as well.

The wireless communication system 208 may be any system configured towirelessly couple to one or more other vehicles, sensors, or otherentities, either directly or via a communication network as describedabove with respect to the wireless communication system 152 in FIG. 1.While the wireless communication system 208 is shown to be positioned ona roof of the vehicle 200, in other examples the wireless communicationsystem 208 could be located, fully or in part, elsewhere.

The camera 210 may be any camera (e.g., a still camera, a video camera,etc.) or image-capture device configured to capture images of theenvironment in which the vehicle 200 is located. To this end, the camera210 may take any of the forms described above with respect to the camera134 in FIG. 1. While the camera 210 is shown to be mounted inside afront windshield of the vehicle 200, in other examples the camera 210may be mounted elsewhere on the vehicle 200, either inside or outsidethe vehicle 200.

The vehicle 200 may include one or more other components in addition toor instead of those shown.

A control system of the vehicle 200 may be configured to control thevehicle 200 in accordance with a control strategy from among multiplepossible control strategies. The control system may be configured toreceive information from sensors coupled to the vehicle 200 (on or offthe vehicle 200), modify the control strategy (and an associated drivingbehavior) based on the information, and control the vehicle 200 inaccordance with the modified control strategy. The control systemfurther may be configured to monitor the information received from thesensors, and continuously evaluate driving conditions; and also may beconfigured to modify the control strategy and driving behavior based onchanges in the driving conditions.

II. Example Method

FIG. 3 is a flow chart of a method 300 for using detected objects forimage processing. Other example methods for using detected objects forimage processing may exist as well.

The method 300 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 302-310. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

In addition, for the method 300 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ormemory, for example, such as a storage device including a disk or harddrive. The computer readable medium may include a non-transitorycomputer readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and Random Access Memory (RAM). The computer readablemedium may also include non-transitory media or memory, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, a tangible storagedevice, or other article of manufacture, for example.

In addition, for the method 300 and other processes and methodsdisclosed herein, each block in FIG. 3 may represent circuitry that iswired to perform the specific logical functions in the process.

At block 302, the method 300 includes receiving, from an image-capturingdevice coupled to a vehicle, an image comprising an object. In someimplementations, a computing device may be configured to receive imagesfrom an image-capturing device. The computing device may be currentlycontrolling a vehicle or may be configured to control a vehicleautonomously or some of the systems of the vehicle, such as the examplesshown in FIGS. 1-2. Further, the computing device may receive imageswirelessly or through a wired route. In other examples, other devicesmay receive the images from an image-capturing device. The images may bestored in various types of memory or within a cloud.

The images captured by an image-capturing device may display particularranges of the environment of the vehicle, or aimed at particularobjects. In some implementations, the images may depend on the placementand orientation of the image-capture device on the vehicle. The imagesmay include multiple objects or may focus upon a single object. Inaddition, the images may vary in size and quality and may betwo-dimensional or three-dimensional. Similarly, the computing devicemay receive the images in real-time. The images may be stored in memoryor may be volatile, existing only for a period of time.

An image-capture device may be a camera, video recorder, or any type ofdevice capable of capturing images. The image-capture device may beconfigured to record or capture images that can be stored directly,transmitted to another location, or both. In addition, the image-capturemay be configured to capture still photographs or moving images, such asin videos. The image-capturing device may be part of a system associatedwith a computing device and may be coupled to the vehicle in a varietyof places on the vehicle. For example, one or more image-capture devicesmay be built into the bumper of the vehicle or may be part of thewindshield.

The computing device controlling a vehicle autonomously may continuouslyreceive images from a system of image-capture devices or mayperiodically receive the images. The images may be used to analyze theenvironment surrounding the vehicle, including reading signs, analyzingpotential hazards and objects, and other purposes.

At block 304, the method 300 additionally includes receiving, via one ormore sources coupled to the vehicle, information indicating one or morecharacteristics of the object. A computing device associated with avehicle may be configured to use additional sources to receiveinformation indicating characteristics of objects in the vicinity of thevehicle. The computing device may be the same computing device receivingthe images as discussed above or a different computing device that maybe in communication with the computing device controlling the vehicle.

Examples of sources may include RADAR, LIDAR, additional cameras, maps,GPS, sensors, or additional sources. The sources may be coupled to thevehicle or may be associated with the vehicle without a physicalconnection. The sources may transmit information wirelessly or through awired-connection. Further, the sources may be configured to work in asystem or may be configured to operate in separate systems. A computingdevice may be configured to all of the sources, a portion of thesources, or none of the sources. Different computing devices may controldifferent sources. The sources may communicate via a wired-link,wirelessly, or within a cloud, for example. In addition, differentsources may be configured to determine different characteristics of theobjects in the surrounding environment of a vehicle.

The computing device may receive information indicating characteristicsof objects nearby the vehicle from one or more sources. For example, acomputing device may receive information, such as whether an object ismoving or stationary, a longitudinal speed of the object, a lateralspeed of the object, and the direction of motion of the object and/or asize of the object. In addition, a computing device may receiveinformation about the respective position of the object on a road orother surface which the vehicle is traveling.

In some implementations, additional sources coupled to an autonomousvehicle may provide the computing device with perspective and factors toanalyze about the environment and objects surrounding the vehicle. Theinformation from the additional sources may be used during imageprocessing. In additional examples, a computing device may also receiveinformation about the environment, such as weather information or anoverall traffic status on a specific road. A computing device mayreceive additional information from sources about the environment orobjects within the environment as well.

In one implementation, a computing device may be configured to receiveinformation from lasers coupled to the vehicle. The LIDAR may beconfigured to emit light through a process of optical amplificationbased on the stimulated emission of electromagnetic radiation. Further,a computing device may use LIDAR to mark target objects, measure ranges,and speeds or for other purposes. LIDAR may be useful to determinedistances between a vehicle and objects. Other information may begathered by a computing device using LIDAR as well. In addition, acomputing device may use LIDAR or a similar system as discussed above todetermine information about the characteristics one or more objectswithin the vicinity of the vehicle.

Similarly, a computing device may be configured to use RADAR in additionto other sources to gather information about an object. As discussedabove, a computing device controlling a vehicle autonomously may useRADAR to determine information on an object, such as a range, altitude,direction, speed, or other information associated with the object. Inaddition, RADAR may be used by a computing device rather than othertypes of sources. Further, a computing device may receive additionalinformation about an object from a GPS system associated with thevehicle. GPS may be used to gather location information about objectsrelative to the position of the vehicle. A computing device may receivea combination of information from multiple sources, such as RADAR andGPS.

In some implementations, a computing device may use additional camerasto gain information about an object within the vicinity of the vehicle.For example, a computing device may use cameras that are positioned atdifferent points on the vehicle to capture images of an object fromdifferent angles. In addition, a computing device may use cameras thatare configured to take more or less focused images, which may providevariances for analyzing a main image. The various cameras may havedifferent lens.

In other implementations, a computing device controlling a vehicleautonomously may use other types of sensors or sources to receiveinformation about the surrounding environment of the vehicle, includingmore specific information about objects or the environmental in general.

At block 306, the method 300 further includes determining by a computingdevice, using the information indicating one or more characteristics ofthe object, a portion of the image depicting the object. A computingdevice controlling a vehicle autonomously may process the imagesreceived from the image-capture device in order to develop a controlstrategy. Processing the images may include using one or more transformsto map locations from three-dimension objects in the environment to aproportional two-dimension format. The images may contain some objectsof interest to the computing device and thus, a computing device may beconfigured to focus upon the objects within the images for furtheranalysis. In some implementations, the computing device may use theinformation received from the various sources to further process theimages and focus upon portions of the image with the objects ofinterest.

In some implementations, a computing device may be configured to focusupon particular portions of received images that contain objects ofinterest to the computing device. The computing device may be able tofurther analyze an image of the environment by locating objects ofinterest within the image. An autonomous vehicle may adjust a controlstrategy based on objects within the local environment of the vehicle.For example, the autonomous vehicle may capture an image of a trafficsignal. The autonomous vehicle may process the image and determine thatthe vehicle may need to stop in order to obey the current state of thetraffic signal.

In some examples, the computing device may receive information fromsources about nearby objects prior to receiving images from animage-capture device. Similarly, the computing device may receive imagesprior to receiving information about objects within the image. Thus, thecomputing device may process images to determine information aboutsurroundings, including objects in the vicinity of the vehicle, whichmay be used for operating autonomously.

In some implementations, a computing device may use the informationabout nearby objects received from the sources to process images. Forexample, the computing device may use information from the sources todetermine the approximate locations of the objects within the image. Thecomputing device may use the information received from sources to focusupon a portion or portions of the image that contain the objects ofinterest to the computing device. For example, the computing device mayfactor the characteristics about the size of an object relative tosurrounding entities during determining the location of the objectwithin the image. The portion of the image may be the same size of theimage captured by the image-capture device or may be smaller than theimage. A computing device may use the various information received fromthe different types of sources to focus upon objects in the image ordifferent portions of the image. The combination of information frommultiple sources may provide a computing device with more options andinformation than information from a single source, such as in an image.

In some implementations, the computing device may determine a portion ofthe image depicting a specific object by determining that one or morepixels within the image match information indicating characteristics ofthe object received from the sourced coupled to the vehicle. Inaddition, the computing device or another device may be configured toperform a translation of a location of the object in the actualenvironment to one or more corresponding pixels in images received froman image-capture device.

The computing device may use some or all of the information provided bythe different sources to locate an object within an image captured ofthe vicinity of the vehicle quickly. For example, the computing devicemay use the information from other sources to hypothesize approximatelocations for objects within the image captured by the image-capturedevice of the vehicle. A computing device may use a transform betweenthe location of points in the environment and the location of the samepoints within the image taken by the autonomous vehicle to determinewhere each or some of the objects should appear in the image. Thetransform may be a three dimension rotation and/or translation from thereal environment to the coordinates of the image-capture devicecoordinates followed by a projection transform dependent on the focallength of the image-capture device and the resolution of any sensors. Insome instances, the transform may map points of three-dimensionalobjects in the environment to a two-dimensional plane within the image.The transform may apply a camera projection matrix, which describesmapping of a pinhole camera from three-dimensional points in the realenvironment to two-dimensional points in an image. Further, thecomputing device may apply extrinsic calibration through rotation andtranslation and may apply intrinsic transformation through projection.In an example, the computing device may translate or rotate elements inthe images as well. In some instances, the output from the computingdevice using a transform may include data about pixels within the imagethat correspond to objects within the real environment.

In some implementations, a computing device controlling a vehicleautonomously may be configured to determine if some objects are occluded(not visible) in the image based on other known objects and/orinformation received from sources. A computing device analyzing thesurrounding environment of a vehicle through images may capture objectsin front of other objects. The computing device may use the approximateobject locations or other information to determine which objects may beoccluded. For example, the computing device may focus on intersectionsbetween areas of the image that are filled by an object and determinewhich object should appear in front of the other in such intersectionsin cases where multiple objects may overlap in a two-dimensional image.A computing device may also use occlusion reasoning to detect partialocclusions. For example, a computing device may determine that the leftside of a car is likely occluded in an image since the right side of thevehicle is visible.

Thus, a computing device may be configured to use information receivedfrom sources other than the image-capture device to process images takenby the image-capture device of the surrounding environment of thevehicle.

At block 308, the method 300 includes processing the portion of theimage to determine a control strategy for the vehicle. The computingdevice of vehicle may be configured to process the portion of the imagein order to determine a control strategy for the vehicle. The computingdevice may determine a control strategy based on the objects in thevicinity of the vehicle. For example, the computing device controlling avehicle may determine that another vehicle traveling in front of thevehicle is braking and in response, either brake or change lanes toavoid a collision. During processing the portion of the image todetermine a control strategy, the computing device may determine one ormore features of objects using the information received from sources andimages. For example, the computing device may determine if a nearbyvehicle is using brake signals or turn signals through processing theimage. In addition, the computing device may use information from thesources with or without the image to determine the color of an object ora license plate number, for example. Other example features may bedetermined as well.

A control strategy may represent the future implementations the vehicleand may comprise various instructions or rules associated with trafficinteraction in various driving contexts. The control strategy, forexample, may comprise rules that determine a speed of the vehicle,steering angle, and a lane that the vehicle may travel on while takinginto account safety and traffic rules and concerns (e.g., vehiclesstopped at an intersection and windows-of-opportunity in yieldsituation, lane tracking, speed control, distance from other vehicles onthe road, passing other vehicles, and queuing in stop-and-go traffic,and avoiding areas that may result in unsafe behavior such asoncoming-traffic lanes, etc.).

In an example, a given control strategy may comprise a program orcomputer instructions that characterize actuators controlling thevehicle (e.g., throttle, steering gear, brake, accelerator, ortransmission shifter) based on the modified trajectory. The givencontrol strategy may include action sets ranked by priority, and theaction sets may include alternative actions that the vehicle may beconfigured to take to accomplish a task (e.g., driving from one locationto another). The alternative actions may be ranked based on the modifiedtrajectory, and respective weights assigned to a respective lateraldistance determined with respect to a given object, for example.

In another example, multiple control strategies (e.g., programs) maycontinuously propose actions to the computing device. The computingdevice may be configured to decide which strategy may be selected basedon a weighted set of goals (e.g., maintaining determined lateraldistances, safety, speed, smoothness of the modified trajectory, etc.),for example. Based on an evaluation of the weighted set of goals, thecomputing device, for example, may be configured to rank the multiplecontrol strategies and respective action sets and determine a givencontrol strategy and a respective action set based on the ranking. Usingimage processing and information on detected objects, a computing devicemay effectively determine safe control strategies for real-timenavigation.

At block 310, the method 300 includes providing instructions to controlthe vehicle based on the control strategy for the vehicle. The controlsystem of the vehicle may comprise multiple control strategies that maybe predetermined or adaptive to changes in a driving environment of thevehicle. The computing device may store instructions to control thevehicle in various types of memories or within a cloud.

The computing device may provide the instructions wirelessly or througha wired-link. In addition, the instructions may be configured in variousformats and may be executed by different types of processors anddevices.

In some implementations, a computing device performing the method 300may further determine, based on the portion of the image depicting theobject, one or more features of the object. For example, the computingdevice may determine the location of a turn signal of another vehiclewithin an image and further determine if the turn signal is in use.Similarly, the computing device may determine the color of an object,size of an object, or a license plate number based on using othersources and a captured image. For example, a computing device maycapture an image of a nearby vehicle and further determine usinginformation received from other sources, the model of the vehicle orother information about the nearby vehicle. A computing device may alsouse other sources to determine the information on signs or to recognizethe placement of traffic signals or other objects in the images, forexample. For example, LIDAR coupled to the vehicle may be configured todetect distances between the vehicle and objects, assisting in thecomputing device in determining approximate locations for the objectsrelative to the vehicle in the images captured by the image-capturedevice. Similarly, the computing device may use process the image anduse the distance information provided by the LIDAR in determining acontrol strategy.

A computing device performing the method 300 may extract image-basedfeatures of objects without relying only upon images. The informationfrom other sources may allow the computing device to quickly determinethe approximate locations of objects within an image and may lower theamount of processing power required from the computing device. In oneexample implementation, the computing device may be configured to devotemore resources or focus upon a smaller fraction of the image, such as anintersection containing multiple objects. The computing device mayfurther focus upon smaller portions of the already fraction of the imageto further determine information about objects within the originalimage.

In one implementation, a computing device may use other known objects toestimate or determine which objects may be included in an image capturedof the surrounding environment of the vehicle. In the example, a devicemay project three-dimensional points into an image received from animage-capture device in order to determine if objects may be in front orbehind each other. For example, a computing device may determine that atraffic light 100 meters in front of the vehicle controlled by thecomputing device and a vehicle 20 meters ahead may overlap so that thevehicle 20 meters ahead is blocking portion of the traffic light. Usingsome form of occlusion reasoning, a computing device may determine thatthe closer object (vehicle 20 meters ahead) should be the one in theimage. The computing device may focus on intersections to determinewhich objects should appear in the front within images.

III. Example Implementations

FIGS. 4A-4D illustrate various example scenarios that a computing devicein control of a vehicle may execute method 300 or a similar method forusing detected objects for image processing. The example scenarios shownin the FIGS. 4A-4D serve merely as illustrations and may vary indifferent implementations. Similarly, other example situations orscenarios may exist as well. In the various examples shown in FIGS.4A-4D, a computing device associated with a vehicle, whether in controlor assisting the driver, may use one or more image-capture devices tocapture images of the relative environment of the vehicle. For example,the image-capture device may be built into the grill or front plate ofthe vehicle and capture images of the space including objects within thefront path of the vehicle. Similarly, the image-capture devices may beplaced in various places on the vehicle and may be configured to capturedifferent portions of the environment around the vehicle. The vehiclesdiscussed above in FIGS. 1-2 may also be used.

Although the computing device may be capable of determining objectswithin an image received from an image-capture device, the process mayrequire time, and/or resources from the computing device or otherdevices. Thus, as discussed above, the computing device may useadditional information received from the other sources coupled orassociated with the vehicle, such as RADAR, LIDAR, other cameras, GPS,or sensors, in order to process the image. The computing device mayreceive information about various characteristics of the objects or theenvironment surrounding the vehicle from the other sources. For example,the computing device may receive information indicating whether anobject within the vicinity of the vehicle is moving or stationary, alongitudinal speed of the object, a lateral speed of the object, adirection of motion of the object, a size of the object, and/or arespective position of the object on the road of which the vehicle isalso traveling along. In addition, the computing device may determineother information about an object. Based on the information receivedfrom other sources, the computing device may process images receivedfrom the image-capture device. The computing device may be configureduse the characteristics about objects to be able to determine a portionor portions of the image depicting the objects for the computing deviceto analyze. For example, the computing device may use the assisted imageprocessing to find the visual features within the image that revealinformation about objects. A computing device may use image processingto locate brake or turn signals, identify object colors or models, orother information. Similarly, the computing device may be configured todetermine approximate locations of objects within the image based oninformation received from other sources coupled to the vehicle.

In some examples, the computing device may receive information from theobjects through a wired-connection or wirelessly.

FIG. 4A illustrates an example scenario that shows a computing device ofa vehicle using additional sources, such as RADAR, LIDAR, sensors, GPSor additional cameras, to determine if a nearby vehicle traveling infront of the vehicle controlled autonomously is using a turn signal. Insimilar scenarios, a computing device may use additional sources toassist in processing images in order to locate the license plate number,the color, or other information from the images about the other vehicle.Other example scenarios may exist as well.

FIG. 4A depicts a vehicle 400 traveling along a two lane road. As theroad extends in the distance, the road also extends into a possibleupcoming turn that provides a route for a vehicle to turn off the twolane road. In the example scenario, the vehicle 400 may continue on theroad straight or may change orientation and turn right onto the new roadahead. For illustration purposes, a vehicle traveling behind the vehicle400 may be controlled autonomously by a computing device. The computingdevice may use one or more sources to detect the vehicle 400 and receiveinformation about the vehicle 400. For example, the computing device mayreceive various characteristics about the vehicle 400, including but notlimited to the speed of vehicle 400, the size of vehicle 400, or therelative distance between vehicle 400 and the autonomous vehiclecontrolled by the computing device. The computing device may receiveother information about the vehicle 400 as well.

As shown in FIG. 4A, the computing device controlling the vehicleautonomously traveling behind vehicle 400 may be configured to captureimages of vehicle 400 and use the information received from additionalsources to process the captured images. For example, the computingdevice may determine the portion of the image that focuses upon the turnsignal of vehicle 400 to determine whether the turn signal is on or off.For illustration purposes, box 402 represents an outline that may be theportion of images that the computing device of the autonomous vehiclebehind vehicle 400 may focus upon in order to determine whether or notvehicle 400 is going to execute a right turn or continue travelingstraight. During processing captured images, the computing device mayuse the additional information received from other sources, such assensors, LIDAR, or GPS, may allow the computing device to approximatelylocate and focus upon the turn signals within images captured of vehicle400. For example, LIDAR may provide the computing device withcharacteristics of the vehicle 400, including the distance betweenvehicle 400 and the autonomous vehicle. Sources such as LIDAR, RADAR,sensors, and GPS may provide information may include the relative sizeor speed of vehicle 400. Other information may be gathered about vehicle400 that may assist during image processing as well.

Indeed, box 402 serves merely as an example within the illustration andmay not be visible in the real world. Further, the size of box 402 mayvary in other examples. A computing device may focus on larger orsmaller portions of images to gather information about other vehicles orobjects. Additional boxes may be shown in other examples representingother portions of an image that a computing device may use additionalsources to focus upon.

Similar to FIG. 4A, FIG. 4B illustrates an additional example scenariothat shows a computing device of a vehicle using additional sources,such as RADAR, LIDAR, sensors, or additional cameras, to determine thata vehicle traveling nearby may be braking or using a turn signal. Otheradditional sources may be used as well.

In the example scenario shown in FIG. 4B, a vehicle 404 may becontrolled by a computing device and operating autonomously. In someimplementations, the computing device of vehicle 404 may be configuredto use one or more additional sources to determine if the nearby vehicle406 in the next lane is using brakes lights 408 or turn signals. Thecomputing device of vehicle 404 may be constantly updating a controlstrategy in order to allow the safe travel of vehicle 404. Therefore, asthe vehicle 404 approaches and/or is passed by vehicle 406, thecomputing device may be configured to determine a proper controlstrategy that reacts to the presence of nearby vehicle 406.

Similar to FIG. 4A, the computing device 404 may capture images using animage-capture device of the surrounding environment, which may includevehicle 406. In order to focus on features or specific objects withinthe images, the computing device may use additional sources to receiveinformation about the environment including vehicle 406. For example,the computing device may receive information from LIDAR or sensors thatprovide a relative distance between vehicle 406 and vehicle 404.Similarly, the computing device may receive other information fromsources, such as the speed or size of vehicle 406, or the relativedistance of vehicle 406 from other points in the environment. Otherinformation may be received by the computing device as well.

Using the information from additional sources, the computing device mayprocess images captured to focus on specific details of vehicle 406. Thecomputing device may be configured to estimate an approximate locationof the brake signals of vehicle 406 within images captured using theinformation about the speed, size, or relative distance received fromsources. By focusing upon a smaller portion of the captured images, thecomputing device may be configured to quickly locate the brake signalson vehicle 406 to determine whether the vehicle 406 is braking. Theinformation gathered from other sources by the computing device mayassist in the processing of images captured of the environmentsurrounding the vehicle controlled by the computing device.

Similar to FIG. 4A, FIG. 4B depicts a box 408 that represents a portionof an image or images that a computing device may use one or moresources to determine. The computing device may be configured to processimages of the vehicle 406 and use other sources to narrow the image to aportion covered by box 408. The computing device may use informationgathered about the size of vehicle 406 or the positioning of the vehicle406 relative to the autonomous vehicle in order to find visual featureswithin the image that correspond to brake lights of vehicle 406. Thecomputing device may use the information within a transformation thatprojects points from the image-capture device into the environment andpoints from the environment into the images. Similarly, the computingdevice may use camera calibration or projection and re-projection todetermine locations of objects within images. The additional informationmay contribute to developing matrices that allow the computing device tomatch objects in the nearby environment to points within the images. Thebox 408 serves merely for illustration reasons and may not be visible inthe real environment.

In some examples, the information received from RADAR or LIDAR may helpthe computing device determine the position of objects within theimages. Other entities or sources connected to the vehicle may assistwith image processing.

FIG. 4C illustrates an additional example scenario that shows acomputing device of a vehicle using additional sources, such as RADAR,LIDAR, maps, or additional cameras, to determine that portions of avehicle traveling nearby has pieces occluded by a fence. FIG. 4C depictsa fence 410 and a vehicle 412 shown behind the fence 410. Portions ofthe vehicle 412 are occluded by the fence 410 from the point of viewshown by the illustration.

In the example, a computing device may be operating a vehicle inautonomous mode and maybe using a system of image-capture devices.Similar to the examples illustrated in FIGS. 4A-4B, a computing devicecontrolling a vehicle nearby the fence 410 and vehicle 412 may useadditional sources to assist in image processing and determining thatportions of vehicle 412 may be occluded by the fence 410.

In the example, an autonomous vehicle is in the vicinity of fence 410and vehicle 412 although the FIG. 4C does not depict the vehicle withinthe image. For positional purposes, the fence 410 may be positionedbetween the autonomous vehicle and the vehicle 412 illustrated in theFIG. 4C. The computing device controlling the autonomous vehicle mayreceive images from an image-capture device that appears as shown inFIG. 4C or similar to the example showing vehicle 412 is behind thefence 410. The computing device may use information gathered from othersources to determine whether the fence 410 should be in front of orbehind the vehicle 412 in images captured. The computing device may usethe information gathered from additional sources to determine at theoverlap intersections which object, the fence 410 or the vehicle 412,should appear in front in images. For example, the computing device maycompare the relative distances between the autonomous vehicle and thefence 410 and the autonomous vehicle and the other vehicle 412. Thecomputing device would find that the fence 410 is closer to theautonomous vehicle and therefore, should appear in front of the vehicle412 within images. Similarly, the autonomous vehicle may use LIDAR todetect the closer object. In other examples, the autonomous vehicle mayuse sensors to detect that the fence 411 is closer.

Other information may be gathered from using the additional sources andimages captured. The computing device may compare intersections withinthe image or use other information from sources to determine the properorder of objects within an image.

In the example, the computing device or another entity may project thethree-dimensional points of the environment onto the image to determinethe location of objects within the image. However, some of the objectsmay be occluded, such as portions of vehicle 412 behind the fence 410.The computing device may project both at intersections and determinewhich object should be shown on top within the image. The computingdevice may use information from LIDAR or sensors that provide that thefence 410 is closer to the autonomous vehicle than the other vehicle412. RADAR may be used in a similar manner to inform the computingdevice that the fence 410 is in front of vehicle 412 from this angle,for example.

In similar situations, a computing device of an autonomous vehicle maydetermine other partial or full occlusions. Other examples may exist aswell.

FIG. 4D illustrates an additional example scenario that shows acomputing device of a vehicle using one or more additional sources tolocate and determine information about a traffic signal. FIG. 4D depictsan autonomous vehicle 414 and a traffic signal 416. In addition, similarto the other FIGS. 4A-4B, FIG. 4D depicts a box 418 that illustrates apossible portion of an image that autonomous vehicle 414 may useadditional sources to focus upon within images taken of the generalenvironment surrounding vehicle 414.

In the example scenario, a computing device may be controlling thevehicle 414 and may use a camera or another image-capture device tocapture images of the environment of the vehicle in real-time asdiscussed above. A number of the images may include the traffic signal416, but may not be focused upon the traffic signal 416 specifically.Thus, the computing device of vehicle 414 may use one or more additionalsources to determine more information about the traffic signal 416.Using the additional information from the sources, the computing devicemay be able to focus upon a portion of the images captured containingthe traffic signal 416 as represented by box 418. The box 418 around thetraffic signal 416 may vary in size and may even focus on a particularsignal. Similarly, the box 418 is for illustration purposes and may notexist in the real environment.

In one implementation, the computing device operating vehicle 414 mayreceive information from the traffic signal 416 wirelessly. Theinformation received wirelessly from the traffic signal 416 may be usedto simplify processing images containing the traffic signal. Forexample, the computing device may be configured to use the informationreceived wirelessly from the traffic signal 416 to determine anapproximate location of the traffic signal 416 within an image capturedfor vehicle. Likewise, the computing device of an autonomous orsemi-autonomous vehicle may receive information wirelessly from otherobjects, such as signs, or other vehicles, etc. that may be used incoordination with image processing.

Based on the image processing, the computing device operating vehicle414 may determine the respective state of the traffic signal 416 andoperate according to a control strategy that is in response to the stateof the traffic signal 416. The computing device may select differentcontrol strategies based on different scenarios.

In further examples, upon determining a state of the traffic signal 416,the vehicle 414 may be configured to determine actions of nearbyvehicles. For example, if the traffic signal 416 turns from green toyellow, the vehicle 414 may be configured to search within capturedimages for indications of nearby vehicles braking, for example. Thus, byusing additional information from other sources, e.g., from a sensordetermining a state of the traffic signal 416, the vehicle 414 may focuson areas within images in which brake lights of vehicles may be present(e.g., such as a certain distance above ground level).

In some examples, a computing device controlling a vehicle may processsome images without additional help from other sources and may set apriority for processing some images with certain objects applyinginformation received from the additional sources. A computing device maydevelop more than one particular control strategy or choose one or morepreviously used control strategies in response to processing imagesthrough the assistance of additional information from sources, such asRADAR, LIDAR, maps, GPS, and other sensors.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A system comprising: a first vehicle having asensor and a camera; and a computing device configured to: receive imagedata and sensor data representing an environment of the first vehicle,wherein the image data is received from the camera and the sensor datais received from the sensor; based on the sensor data, determine aposition of a second vehicle relative to the first vehicle; correlatesensor data representing the position of the second vehicle to a set ofpixels in the image data that depicts the second vehicle; determine thata brake signal of the second vehicle is active based on the set ofpixels in the image data that depicts the second vehicle; and based ondetermining that the brake signal of the second vehicle is active, causethe first vehicle to adaptively perform a control strategy.
 2. Thesystem of claim 1, wherein the camera has a field of view of an exteriorenvironment of the first vehicle.
 3. The system of claim 1, wherein thesensor corresponds to a lidar unit.
 4. The system of claim 1, whereinthe sensor corresponds to a radar unit.
 5. The system of claim 1,wherein the control strategy involves an application of brakes by thefirst vehicle based on the position of the vehicle relative to the firstvehicle.
 6. The system of claim 1, wherein the computing device isfurther configured to: correlate the sensor data representing theposition of the second vehicle to the set of pixels in the image datathat depicts the second vehicle based on a camera projection matrix. 7.The system of claim 1, wherein the computing device is furtherconfigured to: perform a transform to correlate the sensor datarepresenting the position of the second vehicle to the set of pixels inthe image data that depicts the second vehicle.
 8. The system of claim1, wherein the computing device is further configured to: determine thata turn signal of the second vehicle is active based on the set of pixelsin the image data that depicts the second vehicle, wherein the controlstrategy is further based on determining that the turn signal of thesecond vehicle is active.
 9. The system of claim 1, wherein thecomputing device is coupled to the first vehicle.
 10. A methodcomprising: receiving, at a computing device coupled to a first vehicle,image data and sensor data representing an environment of the firstvehicle, wherein the image data is received from a camera coupled to thefirst vehicle and the sensor data is received from a sensor coupled tothe first vehicle; based on the sensor data, determining a position of asecond vehicle relative to the first vehicle; correlating sensor datarepresenting the position of the second vehicle to a set of pixels inthe image data that depicts the second vehicle; determining that a brakesignal of the second vehicle is active based on the set of pixels in theimage data that depicts the second vehicle; and based on determiningthat the brake signal of the second vehicle is active, causing the firstvehicle to adaptively perform a control strategy.
 11. The method ofclaim 10, wherein receiving image data and sensor data representing theenvironment of the first vehicle comprises: receiving image datarepresenting an area of the environment located in front of the firstvehicle from the camera, wherein the camera is coupled to the firstvehicle at a forward-facing orientation.
 12. The method of claim 10,wherein receiving image data and sensor data representing theenvironment of the first vehicle comprises: receiving sensor data from alidar unit or a radar unit.
 13. The method of claim 10, wherein causingthe first vehicle to adaptively perform the control strategy comprises:causing an application of brakes by the first vehicle based on theposition of the vehicle relative to the first vehicle.
 14. The method ofclaim 10, wherein correlating the sensor data representing the positionof the second vehicle to the set of pixels in the image data thatdepicts the second vehicle comprises: correlating the sensor data to theset of pixels based on a camera projection matrix.
 15. The method ofclaim 10, wherein correlating the sensor data representing the positionof the second vehicle to the set of pixels in the image data thatdepicts the second vehicle comprises: performing a transform tocorrelate the sensor data representing the position of the secondvehicle to the set of pixels in the image data that depicts the secondvehicle.
 16. The method of claim 10, further comprising: determiningthat a turn signal of the second vehicle is active based on the set ofpixels in the image data that depicts the second vehicle, wherein thecontrol strategy is further based on determining that the turn signal ofthe second vehicle is active.
 17. The method of claim 10, whereincausing the first vehicle to adaptively perform the control strategycomprises: causing the first vehicle to stop at a location behind thesecond vehicle for at least a predetermined time.
 18. The method ofclaim 10, wherein causing the first vehicle to adaptively perform thecontrol strategy comprises: causing the first vehicle to navigate a patharound the second vehicle.
 19. A non-transitory computer readable mediumhaving stored thereon executable instructions that, upon execution by acomputing system, cause the computing device to perform functionscomprising: receiving, at a computing device coupled to a first vehicle,image data and sensor data representing an environment of the firstvehicle, wherein the image data is received from a camera coupled to thefirst vehicle and the sensor data is received from a sensor coupled tothe first vehicle; based on the sensor data, determining a position of asecond vehicle relative to the first vehicle; correlating sensor datarepresenting the position of the second vehicle to a set of pixels inthe image data that depicts the second vehicle; determining that a brakesignal of the second vehicle is active based on the set of pixels in theimage data that depicts the second vehicle; and based on determiningthat the brake signal of the second vehicle is active, causing the firstvehicle to adaptively perform a control strategy.
 20. The non-transitorycomputer readable medium of claim 19, wherein receiving image data andsensor data representing the environment of the first vehicle comprises:receiving sensor data from a lidar unit or a radar unit.